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Abstract 

This  report  summarizes  the  MITLL-AFRL  MT,  ASR 
and  SLT  systems  and  the  experiments  run  using  them 
during  the  2015  IWSLT  evaluation  campaign.  We  build 
on  the  progress  made  last  year,  applying  those  tech¬ 
niques  to  translating  Chinese  to  English  and  English  to 
Chinese.  ASR  systems  are  also  improved  by  refining 
improvements  developed  last  year.  Finally,  we  combine 
our  ASR  and  MT  systems  to  produce  a  English  to  Chi¬ 
nese  SLT  system. 

In  this  current  draft,  starred  results (*)  indicate  that 
final  results  are  still  pending  and  the  exact  values  may 
need  to  be  adjusted. 

1.  Introduction 

During  the  evaluation  campaign  for  the  2015  In¬ 
ternational  Workshop  on  Spoken  Language  Translation 
(IWSLT  ’15)  [1]  our  experimental  efforts  in  machine 
translation  (MT)  centered  on  1)  the  addition  of  hier¬ 
archical  decoding  systems  2)  reranking  n-best  lists  with 
a  neural  net  encoder-decoder  3)  post-processing  of  un¬ 
known  words  in  translation  output  and  4)  system  com¬ 
bination. 

Experimental  efforts  for  the  automatic  speech  recog¬ 
nition  (ASR)  task  focused  on  using  cutting  edge  neural 
net  techniques  and  the  combination  of  HTK  and  Kaldi- 
based  ASR  systems. 

We  combine  both  efforts  to  produce  a  system  for 
the  spoken  language  translation  (SLT)  task.  Various 
segmentation  and  punctuation  strategies  were  explored. 

This  paper  is  structured  as  follows.  Section  2  presents 
our  work  on  the  MT  task,  and  discusses  each  of  the 
techniques  mentioned  above,  ending  with  a  discussion 
of  submitted  systems.  Our  work  on  the  ASR  task  is  dis¬ 
cussed  in  Section  3.  Finally,  our  work  on  the  SLT  task 
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is  discussed  in  Section  4. 

2.  Machine  Translation 

2.1.  Data  Usage 

Unless  otherwise  noted,  data  described  in  this  sec¬ 
tion  originates  from  the  WMT15  website1.  We  used 
the  parallel  in-domain  data  supplied  by  WIT3  [2].  In 
Chinese-English,  we  additionally  used  the  Yandex  cor¬ 
pus2,  Common  Crawl,  Wiki  Headlines,  News  Crawl,  and 
the  LDC  Gigaworcl  corpus  as  sources  of  monolingual 
English  data  for  language  model  training.  In  English- 
Chinese  we  utilized  the  Chinese  portion  of  the  MultiUN 
corpus  as  an  additional  source  of  language  model  train¬ 
ing  data. 

2.2.  Data  Preprocessing  and  Cleanup 

As  in  past  years,  we  applied  a  cleaning  process  to  the 
training  data  as  previously  described  in  [3] .  Chinese  was 
segmented  with  the  Stanford  Segmenter  [4]  using  both 
Chinese  Treebank  (CTB)  and  Peking  University  (PKU) 
models. 

2.3.  Training 

2.3.1.  Phrase  and  Rule  Table  Training 

We  used  the  default  Moses  scripts  when  training 
phrase  and  rule  tables.  For  Chinese  to  English,  we  in¬ 
creased  the  size  of  the  training  corpus  by  concatenat¬ 
ing  output  from  both  the  CTB  and  PKU  segmentation 
models  while  simply  repeating  the  English  portion  of  the 
corpus.  This  allows  us  to  extract  phrases  for  a  greater 
number  of  phrases  than  one  segmentation  alone.  We 
also  experimented  with  outputting  the  k-best  segmen¬ 
tation  choices  for  a  model  while  repeating  the  English 
portion. 

Phrase  and  rule  tables  are  trained  with  default  Moses 
scripts  or  our  custom  MT  pipeline  driver.  Good- Turing 
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smoothing [5]  was  applied  to  both  rule  and  phrase  tables. 

2.3.2.  Language  Model  Training 

We  reuse  our  BigLM15  from  our  WMT15  shared 
translation  task  submission  [6]  as  our  main  English  ma¬ 
chine  translation  language  model.  The  English  data 
sources  listed  in  Section  2.1  were  used  to  train  a  very 
large  6-gram  language  model.  For  Chinese,  we  take  a 
similar  approach  to  English,  using  the  TED  in-domain 
parallel  training  data  and  the  Chinese  portion  of  the 
MultiUN  corpus,  kenlm  [7]  was  used  to  train  6-gram 
models  in  both  languages.  These  models  were  then  bi¬ 
narized  and  stored  on  local  solid-state  disks  for  each 
machine  in  our  cluster  to  improve  load  time  and  reduce 
fileserver  traffic. 

2.4.  Baseline  MT  System 

Our  system  implements  a  fairly  standard  statistical 
machine  translation  (SMT)  architecture.  It  consists  of 
the  following: 


2.5.  Neural  MT  methods 

2.5.1.  Chinese  to  English 

We  reranked  our  n-best  lists  using  an  end-to-end 
neural  MT  system:  our  own  in-house  Torch7  [14]  im¬ 
plementation  of  Sutskever  et  al  [15] ’s  LSTM  encoder- 
decoder  approach.  This  system  was  trained  using  vary¬ 
ing  amounts  of  out-of-domain  UN  data,  followed  by  train¬ 
ing  on  TED  data.  In  the  following  table,  the  UN  sen¬ 
tences  were  ranked  according  to  bilingual  cross-entropy 
difference  [16]  (using  RNNLM  for  the  language  model 
component)  and  the  top  N  were  chosen  to  pretrain  the 
network.  Once  validation  error  settled  down,  the  net¬ 
works  were  then  trained  over  the  200,000  TED  training 
examples.  The  different  models  that  have  been  trained 
can  be  seen  in  Table  2.5.1.  In  common  among  all  of 
them  were  vocab  selection:  vocab  entries  were  taken  if 
they  appeared  at  least  10  times  in  TED,  or  100  times 


id 

d 

N 

dev  ppl 

BLEU 

0 

4 

2.5M 

27.15 

16.83* 

1 

1 

1M 

30.01 

16.55* 

2 

2 

5M 

25.54 

16.75* 

3 

2 

1M 

28.26 

16.90* 

4 

1 

2.5M 

27.98 

16.65* 

5 

2 

2.5M 

24.21 

16.86* 

Table  1:  Uncased  BLEU  on  tst2013  and  perplexity  on 
dev2010.  d  =  LSTM  depth  and  N  =  cross-entropy  filter 
size  for  UN  data. 


0 

2 

3 

4 

5 

0 

16.83* 

16.90 

16.89 

16.77 

16.93 

2 

16.75* 

17.00 

16.69 

16.96 

3 

16.90* 

16.75 

16.89 

4 

16.65* 

16.84 

5 

16.86* 

Table  2:  Cased  BLEU  on  tst2013  for  combinations  of 
encoder-decoders,  numbered  as  in  Table  1. 

in  UN,  or  5  times  in  TED  and  20  times  in  UN.  Rerank¬ 
ing  results  for  individual  systems,  as  well  as  pairwise 
combinations,  can  be  seen  in  Table  2.5.1. 

Using  the  best  scoring  encoder-decoder  (#3),  and 
the  best  scoring  combination,  we  were  able  to  rerank  the 
n-best  list  from  our  best  hierarchical  moses  system,  and 
achieved  significant  gains.  In  particular,  we  increased 
the  score  from  17.23  to  17.64  cased  BLEU  on  tst2013. 

2.5.2.  English  to  Chinese 

For  the  English  to  Chinese  task,  we  achieved  gains  by 
using  the  Neural  Network  Joint  Model  (NNJM  [17]),  and 
by  reranking  using  GroundHog  [18],  the  Montreal  LISA- 
lab  attention  model  system.  The  NNJM  was  trained  us¬ 
ing  our  own  in-house  implemention  in  Theano  [19].  We 
integrated  NNJM  decoding  into  Moses  as  a  feature  func¬ 
tion,  utilizing  self-normalization  and  precomputation  to 
allow  reasonable  runtimes.  Like  the  Torch  system,  the 
GroundHog  system  was  trained  identically  on  the  Mul¬ 
tiUN  data. 

2.6.  System  combination 

This  year  we  experimented  with  system  combination 
techniques  based  on  Rosti  et  al  [20],  a  well  established 
technique  in  machine  translation.  Our  only  additional 
contribution  was  in  sub-selecting  systems  with  which  to 
perform  system  combination.  Among  our  different  col- 


•  Moses  phrase-based  [8]  or  hierarchical  decoding 
with  the  incremental-search  algorithm  [9] 

•  Stanford  Chinese  character  segmentation  [4] 

•  Hierarchical  mslr  lexical  reordering  [10]  for  phrase- 
based  systems 

•  Minimal  phrase  table  [11] 

•  7-gram  brown-cluster  language  model  with  80  classes 

•  BigLM15  [6]  for  English,  consisting  of  WMT  newscrawl 
data,  europarl,  news  commentary 

•  Drem  optimization  [12] 

•  Recurrent  neural  net  language  model  (RNNLM) 
rescoring  [13] 

•  Named  Entity  Postprocessing  [6] 


En-Zh  System 

char  BLEU 

Baseline 

20.60* 

+  400  Class  Factored  LM 

20.90* 

+  RNNLM 

21.23 

+  NNJM 

21.42 

+  GroundHog  2M 

21.70* 

-/+  GroundHog  4M 

21.85 

Table  3:  English-Chinese  system  additions 


id 

Description 

BLEU 

0 

Hiero,  6-iter  Drem  DevlO,  CLM 

17.00* 

1 

(0)  +  fixed  wide  beam 

17.00* 

2 

(0)  +  bigdev 

17.23 

3 

(2)  +  enc-dec 

17.45 

4 

(2)  +  3-iter  alt  Drem  variation 

17.00* 

5 

(2)  +  Drem  variation 

17.00* 

6 

Hiero  Incsearch,  bigdev,  no  rescoring 

16.50* 

7 

PB,  bigdev,  no  rescoring 

16.50* 

8 

PB,  ted+nyt  CLMs,  enc-dec 

17.08 

Table  4:  Some  notes  on  the  systems:  CLM  =  brown 
cluster  language  model,  bigdev  =  dev2010  +  tst2010 
+  tst2011  +  tst2012.  TED  factored  LM  has  80  classes, 
nyt  (LDC  English  Gigaword)  has  600.  All  used  a  varia¬ 
tion  of  LMs  trained  on  WMT’15  data. 

laborators,  we  managed  to  produce  over  30  systems  with 
400+  decode  outputs.  With  the  goal  of  choosing  only  9, 
we  first  filtered  out  systems  with  scores  less  than  some 
minimum  acceptable  value  (in  our  case,  16.50  uncased 
BLEU  on  tst2013).  Then,  we  constructed  a  distance 
metric  as  1  — BLEU (a:,  y)  and  performed  k-medoids  clus¬ 
tering  to  choose  systems  that  were  sufficiently  different 
from  each  other. 

Table  4  lists  different  systems  used  for  combination, 
and  Table  5  lists  a  sampling  of  combinations  tried  and 
their  case-sensitive  BLEU  scores  on  tst2013. 

2.7.  Unknown  Word  Processing 

As  in  our  WMT15  submission  [6],  we  employed  un¬ 
known  word  post-processing  to  handle  any  unknown 
words  in  the  translation  instead  of  simply  dropping  these 
words.  To  test  the  effectiveness  of  this  approach,  we  de¬ 
code  all  test  sets  where  references  are  available  with  a 
bare-bones  Moses  hierarchical  decoding  system  where 
no  rescoring  features  are  employed.  The  resulting  gains 
measured  in  uncased  BLEU  are  shown  in  Table  6.  We 


Combo  id 

Systems  Used 

tst2013  BLEU 

0 

0+1+2+4+8 

17.62 

1 

0+1+3+5+8 

17.64 

2 

0+1+2+4+5+8 

17.64 

3 

0+1+3+4 

17.66 

4 

0+1+5+8 

17.74 

Table  5:  Top  5  systems  out  of  system  combination 


note  that  the  improvements  in  BLEU  are  smaller  for  the 
Chinese-English  language  pair  when  compared  to  our 
efforts  in  processing  unknown  words  in  other  language 
pairs,  such  as  Russian-English[6],  but  we  feel  that  em¬ 
ploying  these  processes  are  still  worthwhile  due  to  the 
positive  impact  on  readability  of  the  machine  transla¬ 
tion  output.  Our  technique  adapted  to  Chinese-English 
is  described  in  the  following  section. 

2.7.1.  Chinese  to  English  post-processing 

The  named  entity  list  used  for  named  entity  post¬ 
processing  comes  from  manual  translations  of  named 
entities  found  in  train  2014.  It  was  expanded  by  adding 
versions  of  the  Chinese  name  with  the  common  nouns 
stripped  off.  A  list  of  29  typical  common  nouns  end¬ 
ings  of  named  entity  phrases  was  compiled.  Common 
nouns  like:  Dm  (disease),  -H  (county),  jj%  (race/people), 
(laboratory),  (lake),  HH  (corporation), 
(archipelago)  can  sometimes  be  optionally  included  or 
omitted  by  the  speaker  or  optionally  split  off  of  entities 
by  word  segmenters  or  named  entity  taggers. 

The  output  is  searched  for  words  containing  any  Chi¬ 
nese  characters.  Any  unknown  word  consisting  of  a 
single  character  is  deleted  since  single-character  entities 
are  rare  in  this  domain  (and  segmentation  errors  are  a 
more  common  explanation  for  unknown  single-character 
words).  If  the  word  is  not  found  in  the  named  entity 
word  list,  the  list  is  searched  again  for  the  entity  with 
common  nouns  stripped.  Remaining  unknown  words  are 
deleted  from  the  output. 


Test  Set 

base.  BLEU 

post.  BLEU 

A  BLEU 

tst2013 

16.09 

16.19 

+0.10 

tst2012 

13.64 

13.65 

+0.01 

tst2011 

15.21 

15.29 

+0.08 

tst2010 

12.43 

12.50 

+0.07 

Table  6:  NE  post-processing  improvement  measured  in 
uncased  BLEU. 


2.8.  Submission 

Our  primary  Chinese-English  MT  submission  is  sys¬ 
tem  #3  in  Table  4.  We  submitted  system  #4  in  Table 
5  as  contrastive.  For  English-Chinese,  the  primary  sys¬ 
tem  was  the  last  entry  in  Table  3. 

These  systems  were  used  to  decode  the  tst2014  and 
tst2015  test  sets.  Results  from  scoring  performed  by 
the  workshop  organizers  are  listed  in  Table  7  including 
baseline  system  scores  as  determined  by  the  organizers. 


System 

Lang  Pair 

Test  Set 

BLEU 

Baseline 

Zh-En 

tst2014 

11.43 

Primary 

Zh-En 

tst2014 

14.13 

Contrastive 

Zh-En 

tst2014 

13.35 

Baseline 

Zh-En 

tst2015 

13.59 

Primary 

Zh-En 

tst2015 

16.86 

Contrastive 

Zh-En 

tst2015 

15.05 

Baseline 

En-Zh 

tst2014 

17.74 

Primary 

En-Zh 

tst2014 

18.51 

Baseline 

En-Zh 

tst2015 

21.86 

Primary 

En-Zh 

tst2015 

24.31 

Table  7:  Official  results  measured  in  cased  BLEU. 


3.  ASR 

Acoustic  training  data  for  our  ASR  sytems  were  har¬ 
vested  from  1787  TED  talks.  We  applied  the  same  align¬ 
ment  and  closed  caption  filtering  process  as  we  did  in 
IWSLT  2013  [21],  yielding  336  hours  of  audio. 

An  i-vector  system  was  first  developed  on  the  TED 
data  using  Hidden  Markov  Model  Toolkit  (HTK) 3  Mel- 
Frequency  Cepstral  Coefiiect  (MFCC)  features  and  the 
MIT-LL  i-vector  software.  The  elements  of  the  50  di¬ 
mensional  MFCC  vector  were  based  on  those  used  by  the 
ALIZE  toolkit  [22].  Non-speech  frames  were  removed 
using  the  word  alignments  from  the  closed  caption  fil¬ 
tering  process,  and  the  features  were  normalized  to  zero 
mean  and  unit  variance  on  a  per-speaker  basis.  The  uni¬ 
versal  background  model  included  1024  Gaussians  with 
diagonal  covariances,  and  the  i-vector  dimension  was  set 
to  100.  Lastly,  the  Eigen  Factor  Radial  method  [22]  was 
applied  to  normalize  the  i- vectors. 

A  hybrid  deep  neural-net  (DNN)  -  hidden  Markov 
model(HMM)  speech  recognition  system  was  developed 
using  Theano  and  a  version  of  HTK  that  we  modified 
according  to  the  method  of  [23].  A  context  window 
of  9  frames  was  used  on  the  input,  and  the  speaker- 
specific  i-vector  was  appended  to  each  set  of  stacked  fea¬ 
tures  [24] .  The  feature  set  consisted  of  24  log  filterbank 
outputs  with  delta  and  acceleration  coefficients;  the  fea- 

3http:/ /htk. eng. cam. ac.uk 


tures  were  normalized  to  zero  mean  and  unit  variance 
on  a  per-speaker  basis.  The  DNN  included  5  hidden 
layers  with  1024  rectified  linear  units  per  hidden  layer 
and  8000  output  units.  The  network  weights  were  ini¬ 
tialized  as  suggested  in  [25] .  Cross-entropy  training  was 
performed  using  a  minibatch  size  of  512  and  an  initial 
learning  rate  of  0.0005  that  was  adjusted  according  to 
the  QuickNet  newbob  algorithm.4 

LM  data  selection  was  implemented  using  the  same 
procedure  as  our  IWSLT  2014  system.  Interpolated  tri¬ 
gram  and  4-gram  LMs  were  estimated  on  TED,  1/8  of 
Gigaword,  and  1/8  of  News  2007-2014  using  the  SRILM 
Toolkit.  A  RNN  maximum  entropy  LM  was  estimated 
on  the  same  set  of  training  texts  using  the  RNNLM 
Toolkit.  The  network  included  160  hidden  units,  300 
classes  in  the  output  layer,  4-gram  features  for  the  direct 
connections,  and  a  hash  size  of  109.  The  LM  vocabulary 
included  100000  words. 

Automatic  segmentation  of  the  test  data  was  per¬ 
formed  using  the  same  procedure  as  in  IWSLT  2014[3], 
except  that  we  padded  the  speech  end  points  by  0.25 
seconds  (instead  of  0.15  seconds).  Recognition  lattices 
were  produced  using  HDecode  with  the  trigram  LM  and 
then  rescored  with  the  4-gram  LM.  Next,  1000-best  lists 
were  extracted  from  each  lattice  and  rescored  with  the 
RNN  LM.  The  final  LM  scores  were  obtained  by  linearly 
interpolating  the  log  probabilites  from  the  4-gram  and 
RNN  LM.  Interpolation  weights  of  0.25  for  the  4-gram 
and  0.75  for  the  RNN  were  chosen  based  on  results  from 
previous  experiments. 

Adaptation  data  was  selected  for  each  speaker  us¬ 
ing  confidence  scores  [26].  In  our  work,  we  estimated 
confidence  scores  at  the  acoustic  frame  level  by  aligning 
the  20-best  hypotheses  for  each  utterance  and  count¬ 
ing  the  number  of  matching  HMM  shared  states.  Next, 
speaker-dependent  DNNs  were  estimated  on  frames  with 
a  confidence  score  of  0.9  or  higher.  For  each  speaker, 
the  initial  DNN  was  updated  using  a  learning  rate  of 
0.0000625  and  a  single  epoch  of  training.  The  test  set 
was  then  decoded  a  second  time  and  LM  rescoring  was 
reapplied. 

A  second  ASR  system  was  built  using  the  Kaldi  open 
source  speech  recognition  toolkit  [27].  This  system  was 
based  on  the  LIUM  recipe  as  released  with  Kaldi  un¬ 
der  egs/tedlium/s5.  The  details  of  the  particular  sys¬ 
tem  used  for  the  IWSLT  2015  Kaldi-based  ASR  sys¬ 
tem  are  as  follows.  The  acoustic  model  training  data 
and  LM  data  matched  exactly  what  was  used  as  previ¬ 
ously  described  in  the  HTK  ASR  system.  The  first  step 
was  to  build  a  network  to  produce  bottleneck  (BN)  fea¬ 
tures  [28].  MFCCs  from  40  filterbanks  and  3  pitch  fea¬ 
tures  were  used  as  input  to  a  neural  network  of  2  hidden 
layers  each  of  dimension  1500  with  a  40  dimension  BN 
layer  producing  the  output  features.  These  40  BN  fea- 

4 http:/ /www. icsi.berkeley.edu/Speech/faq/nn-train.html 


ASR  System 

Decode 

4-gram 

4-gram+RNN 

HTK  first-pass 

13.7 

13.0 

11.9 

HTK 

11.3 

10.9 

10.0 

Kaldi 

13.3 

12.6 

11.4 

Table  8:  English  tst2013  WER. 


tures  were  then  used  to  build  a  GMM-HMM.  Speaker 
adaptive  training  was  then  conducted  on  this  GMM- 
HMM  using  feature-space  maximum  likelihood  linear  re¬ 
gression  (fMLLR)  transforms.  These  models  were  then 
used  to  train  a  DNN  of  the  Deep  Belief  Network  (DBN) 
variety  described  as  having  6  hidden  layers  with  2048 
neurons  per  layer.  Four  additional  iterations  using  the 
state-level  Minimum  Bayes  Risk  (sMBR)  discriminative 
crierion  were  then  executed.  This  system  was  evaluated 
using  the  trigram  LM  to  produce  recognition  lattices, 
which  were  then  rescored  with  the  4-gram  and  RNN 
LMs  as  described  for  the  HTK  system. 

Table  8  shows  the  WER  of  each  system  on  tst2013 
after  evaluating  the  decoder,  rescoring  with  the  4-gram 
LM,  and  interpolating  the  4-gram  and  RNN  LM  scores. 
For  comparison  purposes,  we  included  the  results  of  the 
HTK  system  prior  to  updating  the  weights  of  the  DNN 
(denoted  as  HTK  first-pass).  The  final  hypothesis  was 
selected  by  applying  N-best  ROVER  to  the  output  from 
the  HTK  system  and  the  Kaldi  system.  This  yielded  a 
9.4%  WER  on  tst2013  and  a  7.4%  WER  on  tst2015. 

4.  SLT 

New  for  this  year,  we  combine  our  efforts  in  ASR 
and  MT  to  produce  an  entry  to  the  SLT  task  for  the 
English-Chinese  language  pair.  We  use  the  rover  out¬ 
put  from  system  combination  from  the  ASR  task  and 
translate  it  with  a  variant  of  our  best  English-Chinese 
MT  system.  For  segmenting  the  output,  we  used  the 
segmentations  produced  by  the  ASR  system,  based  on 
lengths  of  pauses.  To  repunctuate  the  ASR  output,  we 
created  a  classifier  based  on  a  recurrent  neural  network. 
For  each  word,  the  classifier  reports  which  punctuation, 
if  any,  follows  it.  The  output  layer  is  a  softmax  over  a 
limit  set  (period,  comma,  question  mark,  exclamation 
point,  and  no  punctuation).  The  inputs  to  the  classi¬ 
fier  are  a  gated  recurrent  unit  [29]  hidden  state  for  the 
word  in  question,  as  well  as  its  word  vector  and  the  word 
vectors  for  the  following  two  words  after.  Our  experi¬ 
mentation  was  quite  limited,  but  we  observed  that  (a) 
having  the  word  vector  as  well  as  the  recurrent  state  for 
the  current  word  was  helpful,  and  (b)  three  layer  deep 
gated  recurrent  unit  worked  best  out  of  1-4.  The  system 
was  trained  on  the  English  side  of  TED  data,  with  600- 
dimensional  word  vector  size,  and  vocabulary  of  about 
60K  words.  We  did  not  try  any  other  repunctuation 
techniques. 


One  of  our  alternate  approaches  for  adapting  ASR 
output  to  MT  involves  taking  the  output  of  the  ASR  sys¬ 
tem  when  decoding  dev2010  then  using  the  mwerAlign[30] 
program  to  then  fit  the  ASR  output  segments  to  the  En¬ 
glish  portion  of  the  dev2010  tuning  set.  We  then  tune 
the  MT  system  with  this  new  dev  set  in  order  to  better 
match  the  ASR  English  output  to  the  English-Chinese 
MT  system.  We  submitted  this  as  a  contrastive  system. 
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